t • 



PATENT APPLICATION SERIAL NO. 



U.S. DEPARTMENT OF COMMERCE 
PATENT AND TRADEMARK OFFICE 
FEE RECORD SHEET 



IS- 



08/31/1999 HVILLARI 00000047 500417 09374041 

01 FC:108 760.00 CH 

02 FC:110 16.00 CH 

10/27/2000 JMSHING 00000013 500417 09374041 
01 FC:105 130.00 CH 



PTO-1556 
(5/87) 



*U S. GPO: 1998-433-214/80404 



50 



sketNo.: 50246-068 



vol 



PATENT 



: H 



IN THE UNITED STATES PATENT AND TRADEMARK OFFICE ] £ 



in re Reissue of U.S. Patent 5,598,525 
Issued January 28, 1997 
To Robert M. Nally, et al. 
Based on Serial No.: 376,919 
Filed: January 23, 1995 




'CO 



Group Art Unit: 
Examiner: 



For: APPARATUS, SYSTEMS AND METHODS FOR CONTROLLING GRAPHICS AND VIDEO 
DATA IN MULTIMEDIA DATA PROCESSING AND DISPLAY SYSTEMS 



Stat 

~* Assistant Commissioner for Patents 
Washington, D.C. 20231 



REISSUE APPLICATION TRANSMITTAL 



ox7 



y3 



=gir: 

^Transmitted herewith is an application for reissue of U.S. Patent No. 5598525 issued on January 23, 1995 to 
ypventors Robert M. Nally and John C. Schafer. 

m 



0 ^pnclosed are the following: 



1*. SPECIFICATION, CLAIMS AND DRAWINGS 



(a) 



(b) 



□ 
□ 



□ 



14 pages of specification and cover page 
9 pages of claims 
1 page of abstract 
sheets of formal drawings 

No changes in the drawings upon which the original patent was issued are to be 

made. Therefore, in accordance with 37 CFR §1.174, please find attached, in the 

size required for original drawings: 

a copy of the printed drawings of the patent (4 sheets) 

a photoprint of the original drawings. 



2. DECLARATION AND POWER OF ATTORNEY 

^ 12 pages of declaration and power of attorney (unsigned) 



PRELIMINARY AMENDMENT 
attached 

OFFER TO SURRENDER THE ORIGINAL LETTERS PATENT IN ACCORDANCE WITH 37 
CFR §1.178 

^ attached (unsigned) 

LETTERS PATENT 

□ original letters patent attached 

□ declaration that original letters patent lost or inaccessible 

TITLE 

In accordance with 37 CFR §1.171, this application for reissue is accompanied by: 
Q a certified copy of an abstract of title 
or 

X Certificate under 37 C.F.R. 3.73(b) 

INFORMATION DISCLOSURE STATEMENT 
E>3 attached with 21 Exhibits 

PRIORITY -35 USC §119 

□ Priority of Patent Application filed on in is claimed under 35 USC §119. 

□ The certified copies have been filed in prior application Serial No. filed on . 

FEE CALCULATION (37 CFR §1.16(i) AND (j) 
Claims as Filed: 





NO. OF 
CLAIMS 




EXTRA 
CLAIMS 


RATE 


AMOUNT 


$<ttal Claims 


48 


-20 




$18.00 = 


$504.00 


jiipependent Claims 


6 


-3 




$78.00 = 


$234.00 


— ^ — . 

Basic Application Fee 


$790.00 


Fee for Title Report 


0 


Total Application Fee 


$1528.00 


Subtract Vi if small entity 




TOTAL APPLICATION FEE DUE 


$1528.00 







1 0. SMALL ENTITY STATUS (if applicable) 

□ A verified statement that this filing is by a small entity is attached 



2 



4. 



5. 



6. 



ltd iaa 
Ft 

u sc 

i'3- SF 

si: 

r m 



11. 



12. 



13. 



U 



'1 ] 

in 



Is** 



,5 3=3* 



ASSIGNMENT INFORMATION (if applicable) 

^ The prior application is assigned of record to . Cirrus Logic, Inc. See Certificate 
under 37 CFR. 3.73(b), attached. 

METHOD OF PAYMENT OF FEES 

^ Charge Deposit Account No. 500417 in the amount of $ 1510.00_. Two copies of 
this request are attached. 

AUTHORIZATION TO CHARGE ADDITIONAL FEES 

^ The Commissioner is hereby authorized to charge payment of the following fees 
associated with this communication and during the entire pendency of this 
application or credit any overpayment to Deposit Account No. 500417. Two 
duplicate copies are enclosed. 

IE 37 CFR §1.16 (filing fees) 

^ 37 CFR §1.16 (presentation of extra claims) 

£3 37 CFR §1.1 6(e) (surcharge for filing the basic filing fee and/or declaration on a date 
later than the filing date of the application. 
37 CFR §1.17 (application processing fees) 

37 CFR §1. 18 (issue fee at or before mailing of Notice of Allowance, Pursuant to 37 
CFR 1. §3 11(b). 

Respectfully submitted, 
MCDERMOTT, WILL & EMERY 

David L. Stewart 
Registration No. 37,578 



$00 13 th Street, N.W. 
Washington, DC 20005-3096 
l#)2) 756-8000 DLS:pgb 
Date: August 13, 1999 
Facsimile: (202) 756-8087 



3 



1 



APPARATUS, SYSTEMS AND METHODS 
FOR CONTROLLING GRAPHICS AND 
VIDEO DATA IN MULTIMEDIA DATA 

PROCESSING AND DISPLAY SYSTEMS 

5 

TECHNICAL FIELD OF THE INVENTION 

The present invention relates in general to multimedia 
processing and display systems and in particular to appara- 
tus, systems and methods for controlling graphics and video \q 
data overlay in multimedia processing and display systems. 

CROSS-REFERENCE TO RELATED 
APPLICATIONS 

The following copending and coassigned United Stales 15 
patent applications contain related information and are 
incorporated herein by reference: 

U.S. patent application Ser. No. 08/098*846 (Attorney's 
Docket No. P3510-P11US), entitled "System And Method 2 a 
For The Mixing Of Graphics And Video Signals," and filed 
Jul. 29, 1993; and 

U.S. patent application Ser. No. 08/223,845 (Attorney's 
Docket No. P3510-P21US), entitled "Apparatus, Systems 
And Methods For Processing Video Data In Conjunction 25 
With A Multi-Format Frame Buffer " and filed Apr. 6, 1994. 

BACKGROUND OF THE INVENTION 

As multimedia information processing systems increase 30 
in popularity, system designers must consider new tech- 
niques for. controlling the processing and display of data 
simultaneously generated by multiple sources. In particular, 
there has been substantial demand for processing systems 
which have the capability of concurrently displaying both 35 
video and graphics data on a single display screen. The 
development of such systems presents a number of design 
challenges, not only because the format differences between 
graphics and video data must be accounted for, but also 
because of end user driven requirements that these systems ^ 
allow for flexible manipulation of the data on the display 
screen. 

One particular technique for simultaneously displaying 
video and graphics data on a single display screen involves 
the generation of "windows." In this case, a stream of data 4s 
from a selected source is used to generate a display within 
a particular region or "window" of the display screen to the 
exclusion of any non-selected data streams defining a dis- 
play or part of a display corresponding to the same region of 
the screen. The selected data stream generating the display 50 
window "overlays" or "occludes" the data from the nonse- 
lected data streams which lie "behind" the displayed data. In 
one instance, the overall content and appearance of the 
display screen is defined by graphics data and one or more 
"video windows" generated by data from a video source 55 
occlude a corresponding region of that graphics data, In 
other instances, a video display or window may be occluded 
or overlaid by graphics data or even another video window. 

In the multimedia environment, the "windowing*' 
described above yields substantial advantages. Among other 60 
things, the user can typically change the size and location on 
the display screen of a given window to flexibly manipulate 
the content and appearance of the data being displayed. For 
example, in the case of combined graphics and video, the 
user can advantageously create custom composite visual 63 
displays by combining multiple video and graphics data 
streams in windowing environment 
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In order to efficiently control windows in a multimedia 
environment efficient frame buffer management is required. 
Specifically, a frame buffer control scheme must be devel- 
oped which allows for the efficient storage and retrieval of 

5 multiple types of data, such as video data and graphics data. 
To be cost competitive as well as functionally efficient, such 
a scheme should minimize the number of memory devices 
and the amount of control circuitry required and should 
insure that data Cow to the display is subjected to minimal 

10 delay notwithstanding data type. 

One of the major difficulties in managing video in a 
combined video and graphics windowing environment 
results from the fact that the video data being received and 
displayed are constantly being updated, typically at a rate of 

15 thirty frames per second. In contrast, the graphics data are 
normally generated once to define the graphics display and 
then remain static until the system CPU change that graphics 
display. Thus, the occlusion (overlay) of video data with 
graphics data requires that the static graphics data "in front 

20 of 1 the video data not be destroyed each time the video 
window is updated A second concern with windowing 
systems operating on both video and graphics data is the 
formatting differences between the video and graphics data 
themselves since video is typically digitized into a YUV 

25 color space while graphics is digitized into an RGB color 
space. Hence, any combination video and graphics window- 
ing system must have the capability of efficiently handling 
data within both the YUV and RGB formats. 
Thus, due to the advantages of windowing, the need has 

30 arisen for efficient and cost effective windowing control 
circuitry. Such windowing circuitry should allow for the 
simultaneous processing of data received from multiple 
sources and in multiple formats. In particular, such window- 
ing control circuitry should be capable of efficiently and 

35 inexpensively controlling the occlusion and/or overlay of 
video and graphics data in a windowing environment 



SUMMARY OF THE INVENTION 

(0 

The principles of the present invention in general provide 
for the flexible control of graphics and video data in a 
display control environment In particular, an entire frame of 
video d rt ^ graphics data, or a combination of both, may be 

i5 stored in on-screen memory and rastered out with the 
generation of the corresponding display screen. A window of 
graphics or video data can then be stored in off-screen 
memory and retrieved when the raster scan generating the 
display reaches the desired position on the display for the 

so video window. The window of data from off-screen memory 
can then be overlayed over the data being rastered out of the 
on-screen memory under one of three conditions. In a first 
mode, pixels from the off-screen memory are rastered only 
when the raster scan has reached the position on the display 

53 selected for the window. In a second mode, a window of data 
is rastered from the off-screen memory when the display 
raster scan has reached the display window position and 
graphics data being rastered from the on-screen memory 
matches a color key. In a third mode, the window data is 

60 rastered out of the off-screen memory when the data being 
output from the on-screen memory matches the color key, 
notwithstanding the position of the raster scan. 

According to a first embodiment of the present invention, 
a graphics and video controller is provided which includes 

65 a dual aperture interface, each word associated with an 
address to a selected one of on-screen and off-screen areas 
of an associated unified frame buffer as either graphics or 
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video pixel data. Circuitry is provided for writing a word of 
the pixel data received by the interface to a one of the 
on-screen and off-screen memory areas corresponding to the 
address associated with the received word. Circuitry is also 
included for selectively retrieving graphics and video data 5 
from the on-screen and off-screen memory areas. A first 
pipeline is provided for processing graphics data retrieved 
from the frame buffer and a second pipeline is provided for 
video processing data retrieved from the frame buffer. 

According to a second embodiment of the present inven- 10 
uon, a controller is provided which includes a dual aperture 
port for receiving video and graphics data, each word of the 
data received with an address associated directing the word 
to be processed as either graphics or video data and off- 
screen memory spaces of a frame buffer. A second port is l5 
included for receiving real -time video data. Circuitry is 
provided for generating an address associated with a 
selected one of the memory spaces for each word of received 
real-time video data. Circuitry is included for writing selec- 
tively the words into the on-screen and off-screen memory 2Q 
spaces of the frame buffer. Circuitry is also provided for 
selectively retrieving the words of data from the on-screen 
and off- screen spaces as data is rastered for driving a display. 
A graphics backend pipeline processes ones of the graphics 
words of data retrieved from the frame buffer. A video ^ 
backend pipeline is provided for processing ones of the 
video words of data retrieved from the frame buffer, the 
circuitry for retrieving always rastering a stream of graphics 
data from the frame buffer to the graphics pipeline and 
rastering video data to the video backend pipeline when a 3Q 
display raster scan reaches a display position of a video 
window. An output selector is included for selecting for 
output between words of data output from the graphics 
backend pipeline and words of data output from the video 
backend pipeline. 35 
According to 

a third embodiment of the present invention, a 
display system is provided which includes first and second 
parallel backend pipelines. A multi-format frame buffer 
memory is included having on-screen and off-screen memo- 
ries each operable to simultaneously store data in graphics 40 
and video formats. A dual aperture port is provided for 
receiving both graphics and video data as directed by an 
address associated with each word of data received Cir- 
cuitry for writing is included for writing a word of video or 
graphics data into a selected one of the on-screen and 45 
off-screen areas of the multi-frame buffer. Memory control 
circuitry controls the transfer of data between the first and 
second [back-end] backend 

pipelines and the frame buffer. The system 
further includes a display unit and overlay control circuitry 
for selecting for output to the display unit between data 50 
provided by the first backend pipeline and data provided by 
the second backend pipeline. 

A fourth embodiment of the present invention comprises 
a display data processing system which includes circuitry for 
writing data into an on-screen space of a frame buffer and 55 
circuitry for writing data into an off-screen space of the 
frame buffer. A video pipeline is provided for processing 
video data output from a selected one of the on-screen and 
off-screen spaces. The video pipeline includes a first first- 
in/ first-out memory for receiving selected pixel data from 60 
the selected space. The video pipeline also includes a second 
first-in/first-out memory disposed in parallel to the first 
first-in/first-out memory for receiving other selected data 
from the selected space in the frame buffer. An interpolator 
is provided as part of the video pipeline for generating 65 
additional data by interpolating data output from the first and 
second first-in/first-out memories. A graphics pipeline is 
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disposed in parallel to the video pipeline Tor processing 
graphics data output from a selected one of Lhe on-screen 
and off-scTeen spaces. Finally, an output selector is provided 
for selecting between data output from the video pipeline 

5 and data output from the graphics pipeline. 

The principles of the present invention allow for the 
construction of circuits and systems with substantial advan- 
tages over the prior art. Among other things, the principles 
of the present invention allow both graphics and video data 

10 to be stored in a single unified frame buffer and retrieved 
therefrom in a number of different ways. For example, a 
combination of graphics and video data may be stored in the 
on-screen memory and simply rastered out during screen 
refresh. In another case, an entire screen of graphics or video 

15 data may be stored in the on-screen memory while a window 
of graphics or video data is stored in the off-screen portion 
of memory. The window data can then be rastered out to 
selectively overlay a portion of the data being rastered out of 
the on-screen memory. The overlay may be controlled by 

20 either window display position with a match of the on- 
screen data being rastered out and a color key, or both. 

The embodiments of the present invention provide for the 
efficient and inexpensive overlay of video and graphics data 
in a windowing environment In particular, the use of color 

25 comparison to determine the overlay of data in a window 
region eliminates the need for precise x- and y-posuion data 
for the location of that window and allows for video crop- 
ping to be performed. Further* the use of graphics data to 
control overlay provides substantial advantages in that 

30 graphics data is less subject to the graininess and noise 
problems often found with video data. Further, the user is 
given total control of overlay operations when keying on 
graphics data because the graphics data is computer gencr- 

^ ated, whereas the video data is captured data. 

BRIEF DESCRIPTION OF THE DRAWINGS 

For a more complete understanding of the present inven- 
tion, and the advantages thereof, reference is now made to 
40 the following descriptions taken in conjunction with the 
accompanying drawings, in which: 

FIG. 1 is a top level functional block diagram of a 
multi-media processing and display system embodying the 
principles of the present invention; 
45 FIG. 2 is a more detailed functional block diagram of the 
VGA controller depicted in FIG. 1; 

FIG. 3 is an expanded functional block diagram of por- 
tions of the controller of FIG. 2 with emphasis on the overlay 
5Q control features; 

FIG. 4A is a detailed functional block diagram of a first 
embodiment of the color comparison circuitry of FIG. 3; 

FIG. 4B is a detailed functional block diagram of a second 
embodiment of the color comparison circuitry of FIG. 3; and 
33 FIG. 5 is a detailed functional block diagram of a selected 
one of the video window position control circuits depicted in 
FIG. 3. 

DESCRIPTION OF THE INVENTION 

60 

FIG. 1 is a high level functional block diagram of a 
multi-media processing and display system 100 operable to 
process and simultaneously display on a single display 
screen both graphics and video data according to the prin- 
65 ciples of the present invention. Display system 100 includes 
a central processing unit (CPU) 101 which controls the 
overall operation of system 100 and generates graphics data 
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defining graphics images to be displayed. CPU 101 com- 
municates with the remainder of the system discussed below 
via a local bus 103. System 100 also includes a real-time 
video data source 104. A real lime video stream may be 
presented to the system VGA controller 105 in one of two 5 
ways. First, video data source 104 may be coupled to local 
bus 103 and a video data stream introduced through dual 
aperture addresses. In this case, video source 104 will 
directly address the system frame buffer 107. Second, video 
source 104 may be coupled directly to VGA controller 105 
via a dedicated bus 109 or 'Video port." In this instance, 
VGA controller 105 generates the required addresses into 
frame buffer 107. Real-time video source 104 may be, for 
example, a CD ROM unit, a laser disk unit, a videotape unit, 
television cable outlet or other video data source outputting 
video data in a YUV format. CPU 101 operates in conjunc- 15 
lion with a system memory 108 which stores graphics and 
video data on a real-time basis. System memory 108 may be 
for example random access memory (RAM), floppy disk, 
hard disk or other type of storage device. 

A VGA controller 105 embodying the principles of the 20 
present invention is also coupled to local bus 103. VGA 
controller 105 will be discussed in detail below; however, 
VGA controller 105 generally interfaces CPU 101 and video 
source 104 with a display unit 106 and a mulliformat system 
frame buffer 107. Frame buffer memory 107 provides tern- 25 
porary storage of the graphics and video data during pro- 
cessing prior to display on display unit 106. According to the 
principles of the present invention, VGA controller is oper- 
able in selected modes to store graphics and video data 
together in frame buffer 107 in their native formats. In a 3 
preferred embodiment, the frame buffer area is partitioned 
into on-screen memory and off-screen memory. Frame 
buffer 107 is also a "unified" memory in which video or 
graphics data can be stored in either the on-screen or 
off-screen areas. In the preferred embodiment, display unit 35 

106 is a conventional raster scan display device and frame 
buffer 107 is constructed from dynamic random access 
memory devices (DRAMs). 

FIG. 2 is a more detailed functional block diagram of ^ 
VGA controller 105. The primary circuitry blocks of VGA 
controller 105 include video front-end video pipeline 200, 
memory (frame buffer) control circuitry 201, CRT/window 
control circuitry 202, video window control registers 203, 
video [back-end] backend pipeline 204 and graphics 

[back-end] backend pipeline 204 and graphics back-end pipeline 43 
205. VGA controller 105 further includes a CPU interface 
206 for exchanging instructions and data via a PCI or VL 
bus, such as local bus 103 in system 100, with CPU 101. A 
write buffer 207 and conventional graphics controller 208 
allow CPU 101 to directly control data within frame buffer ^ 

107 via memory control circuitry 201. 

In the preferred embodiment of system 100, CPU 101 can 
write video data and/or read and write graphics data to frame 
buffer 107 via CPU interface 206. In particular, CPU 101 can 
direct each pixel to the frame buffer using one of two maps 55 
depending on whether that pixel is a video pixel or a 
graphics pixel. In the preferred embodiment, each word of 
pixel data ("pixel") is associated with one of two addresses, 
one which directs interpolation of the pixel as a video pixel 
through video front-end pipeline 200 and the other which 
directs interpolation of the pixel as a graphics pixel through 
write buffer 207 and graphics controller 208. As a conse- 
quence, either video or graphics pixel data can then be input 
to CPU interface 206 from the PCI/V1 bus through a single 
"dual aperture" port as a function of the selected address. $5 

Data which is input through the video port 211 is 

address- 
free. In this case, video window controls 213 generates the 
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required addresses to either the on-screen memory area or 
the off-screen memory as a function of display location for 
the video window. In the preferred embodiment, window 
controls 213 generate addresses using the same video con- 
5 trol registers 203 used to control retrieval of the video in the 
[back-end] backend pipeline (i.e., the screen x and y position registers 
500 and 501 discussed below in conjunction with FIG. 5). 
When data is being received through both the CPU interface 
206 and the VPORT 211 simultaneously, the data is inter- 
1Q leaved into memory with the two write buffers 207 and 217 
buffering the data such that neither stream is interrupted or 
forced into a wait state at the source component (i.e., bus 103 
or video source 104). 

It should be noted at this point that frame buffer 107 
includes at least two different data areas or spaces to which 
data can be directed bv the given address (either CPU 103 
or controls 213 generated.) 

Each space can simultaneously 
store graphics or video data depending on the selected 
display configuration. The on-screen area corresponds to the 
display screen; each pixel rastered out of a given pixel 
location in the on-screen area defines a corresponding screen 
pixel. The off-screen area is used to store data defining a 
window for selectively overlaying the data from the on- 
screen memory, fonts and other data necessary bv controller 
25 106. Further, as will be discussed [further] below, both graphics 
and video data may be rastered from frame buffer 107 and 
passed through video backend pipeline 204 while only 
^ : ^ graphics data is ever passed through graphics backend 

; ~ M== pipeline 205. 

h l J JJJ 30 According to the principles of the present invention, there 

LiJ! i\¥ are alternate ways of storing and retrieving graphics and 

1 1 Jpj video data from unified frame buffer 107. 

;= :' }il For example, CPU 103 may write a static graphics back- 

~~ 4^ ground into part of the on-screen memory with the remain- 

gjr 35 ing "window" in the on-screen memory area filled with 

^ "|5 playback video data. ''Playback" video data can be either (1 ) 

af f if: live video data input from the VPORT; (2) YUV (video) data 

^ written through interface 206 by CPU 103; or (3) true color 

5 J"" (5:5:5, 5:6:5, or 8:8:8) RGB graphics data (for example 

40 animation graphics data) written in through either the 
*t VPORT or interface 206. Sirnilariy, a playback video back- 

ground and a window of graphics data may be written into 
M the on-screen area. In each of these cases, the data is rastered 

out as the display is without overlav: the video playback data 
4$ is passed through [the video playback data is passed through] 

: ~ -Jih the video backend pipeline 204 as a function of display 

position by controls 202 and the graphics data passed 
through the graphics backend pipeline 250. 
Windows of data retrieved from the off-screen memory 
30 can be retrieved and used to occlude a portion of the data 
being rastered out of the on-screen memory. For example, a 
window of playback data can be stored in the off-screen 
memory and a frame of static graphics data (either true color 
data or indices to CLUT 234) stored in the on-screen 
55 memory. In this case, the static graphics are rastered out of 
the on-screen memory without interruption and passed 
through the graphics backend pipeline 205. The window of 
data in the off-screen memory is rastered out only when the 
display position for the window has been reached .by the 
60 display raster and is passed through video backend pipeline 
204. As discussed below, data from the video backend 
pipeline 204 can then be used to selectively occlude (over- 
lay) the data being output from the graphics backend pipe- 
line 205. A window of static graphics data (true color or 
65 indices to the CLUT 234) can be stored in off-screen 
memory and used [of] to , 

overlay playback video from the 

on-screen memory[, the]. The 

playback video data is passed 



through the video bacJccnd pipeline 204 and the window of 
static graphics data is passed through the graphics backend 
pipeline 205. 

Bit block transfer (BitBLT) circuitry 209 is provided to 
allow blocks of graphics daia within frame buffer 107 to be 5 
u-ansferrcd, such as when a window of graphics data is 
moved on the display screen by a mouse. Digitai-to-analog 
converter (DAC) circuitry 210 provides the requisite analog 
signals for driving display 106 in response to the receipt of 
cither video data from video [back-end] backend 

pipeline 204 or 10 
graphics data from {back-end pipe line] backend pipeline 

208. 

In implementing the operations discussed above, video 
front-end pipeline 200 can receive data from two mutually 
exclusive input paths. First, in the "playback mode," play- 
back (non-real time) data may be received via the PCI bus ^ 
through CPU interface 206, Second, in the "overlay emu- 
lation mode" either real-time or playback video may be 
received through the video port interface 211 (in system 100 
video port interface 211 is coupled to bus 109 when real- 
time data is being received). The selection of video from the 20 
PCI bus or video from video port interface 211 is controlled 
by a multiplexer 212 under the control of bits stored in a 
video front-end pipeline control register within video control 
registers 203. In the playback mode, either CPU 101 or a 
PCI bus master controlling the PCI bus provides the frame 25 
buffer addresses allowing video front-end pipeline 200 to 
map data into the frame buffer separate and apart from the 
graphics data. In the overlay emulation mode, overlay input 
window controls 213 receives framing signals such as 
VSYNC and HSYNC, tracks these sync signals with 30 
counters to determine the start of each new frame and each 
new line, generates the required addresses for the real-time 
video to the frame buffer space using video window position 
data received from window controls 222 (as discussed 
above, in the preferred embodiment, video data is always 35 
retrieved from either the on-screen on off-screen memory 
and passed through video back-end pipeline 204 as a func- 
tion of display position) and thus the position data from 
controls 222 is used to both write data to memory and 
retrieve data therefromD]-^ general, overlay input video 40 
control windows are controlled bv the same registers which 
control the (back-end] backend 

video pipeline 204, although the req- 
uisite counters and comparators are located internal to 
overlay input video control circuitry 213. 

Video front-end pipeline 200 also includes encoding 45 
circuitry 214 that is operable to truncate 16-bit YUV 422 
data into an 8-bit format and then pack four such 8-bit 
encoded words into a single 32-bit word which is then 
written into the video frame buffer space of frame buffer 
105. Conversion circuitry 215 is operable to convert RGB 50 
555 data received from either the CPU interface 206 and the 
PCI bus or VPORTI/F211 into YCrCb (YUV) data prior to 
encoding by encoding circuitry 214. Conversion circuitry 
215 allows graphics data (for example in a 5:5:5 or 5:6:5 
format) to be introduced through the VPORT or graphics 55 
data to be converted, packed and stored in a YUV format in 
the off-screen memory space by CPU 101. For a more 
complete description of encoder 214 and the associated 
decoder 225 of video pipeline 204, reference is now made to 
incorporated copending coassigned application Set No, 60 
08/223,845. The selection and control of the encoding 
circuitry 214 and conversion circuitry 215 is implemented 
through multiplexing circuitries 212 and 216, each of which 
are controlled by bits in the video control registers. F inally, 
video front-end pipeline 200 includes a write buffer/FIFO 6S 
217 which in one embodiment acts as a write buffer and in 
an alternate embodiment acts as a FIFO for the video 
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backend pipeline 204. In embodiments where buffer 217 acts 
as a wmc buffer for then Y, zooming on the [back-end,] backend, 
discussed below is by replication. In embodiments where 
buffer 217 operates as a FIFO, then the VPORT and front 
and end color conversion by converter circuitry 215 are not 
used for writing data to frame buffer 107. 

Memory control circuitry 201 includes an arbiter 218 and 
a memory interface 219. Arbiter 218 prioritizes and 
sequences requests for access to frame buffer 107 received 
from video front-end pipeline 200 T graphics controller 208 
and bit block transfer circuitry 209. Arbiter 218 further 
sequences each of these requests with the refresh of the 
display screen of display 106 under the control of CRT 
controller 202. Memory interface 219 controls the exchange 
of addresses, data, and control signals (such as RAS, CAS 
and read/write enable) to and from frame buffer 107. 

CRT control/video window control circuitry 202 includes 
the CRT controller 220, window arbiter 221, and video 
display window controls 222. CRT controller 202 controls 
the refresh of the screen of display 106 and in particular the 
rastering of data from frame buffer 107 to display unit 107 
through DAC 210. In the preferred embodiment, CRT con- 
troller 220, through arbiter 218 and memory interface 219, 
maintains a constant stream of graphics data into graphics 
backend pipeline 205 from memory; video or playback 
graphics data is rastered out only when a window has been 
reached by the display raster as determined by display 
position controls of window controls 222 (see FIGS. 3 and 
5 and accompanying text) and CRT controller 220. As will 
be discussed in further detail below, the display of windows 
within the display according to the principles of the present 
invention is controlled in part by circuitry 202. 
Video [back-end] backend 

pipeline 204 receives a window of graph- 
ics video data defining a display window from the on-screen 
or off-screen spaces in frame buffer 107 through a pair of 
first-in/first-out memories 223 and 217 (in embodiments 
where buffer 217 is acting as FIFO B). In the preferred 
embodiment, each FIFO receives the data for every other 
display line of data being generated for display on the 
display screen. For example, for a pair of adjacent lines n-1 
and n+1 in memory (although not necessarily adjacent on 
the display) for the display window, FIFO 223 receives the 
data defining window display line n-1 while FIFO 224 
receives the data defining window display line n+1. When 
buffer 217 is used as FIFO B, writes through video front end 
pipeline 200 are made through write buffer I 207 and 
multiplexer 235. Alternatively, if buffer 217 is used as write 
buffer H, then FIFO B is not imolemented and only a single 
stream is processed by video [back-endl backend 

r pipeline 204 (no Y 

interpolation is performed and Y expansion is by replica- 
don). As will be discussed further below (assuming both 
FIFO A and FIFO B are being used), one or more display 
lines, which falls between line n-1 and line n+1, may be 
selectively generated by interpolation. Decoder circuitry 225 
receives two 32-bit packed words (as encoded by encoder 
214), one from each adjacent scan line in memory, from 
FIFOs 223 and 217. Each 32-bit word, which represents four 
YCrCb pixels, is expanded and error diffused by decoder 
225 into four 16-bit YCrCb pixels. In modes where video 
60 data is stored in the frame buffer in standard 555 RGB or 16 
YCrCb data formats, decoder block 225 is bypassed. 

[Back-end] Backend 

video pipeline 204 further includes a Y inter- 
polator 226 and X interpolator 227. In the preferred embodi- 
ment, during Y zooming (expansion) Y interpolator 226 
65 accepts two vertically adjacent 16-bit RGB or YCrCb pixels 
from the decoder 225 and calculates one or more resarapled 
output pixels using a four subpixel granularity. X interpo- 
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l^^r227 during X zooming (expansion) accepts horizons 
tally adjacent pixels from the Y interpolator 226 and calcu- 
lates one or more rcsarnplcd output pixels using a four 
subpixel granularity. For data expansion using line replica- 
tjon, Y interpolator 226 is bypassed. Y interpolator 226 and 5 
X interpolator 227 allow for the resizing of a video display 
window bemg generated from one to four times. 

The output of X interpolator 227 is passed to a color 
convener 228 which converts the YCrCbdata into RGB data 
for delivery to output multiplexer 304. To reiterate, if l0 
graphics data is passed through, video pipeline .convener 228 
is not used. 

IBack-end] Backend 

video circuitry 204 further includes pipeline 
control circuitry 229, overiay control circuitry 230 and 
output multiplexer 231. Pipeline control circuitry 239 con- IS 
trols the reading of data from video FIFOs 223 and 217, 
controls the generation of interpolation coefficients for use 
by X and Y interpolators 226 and 227 to resize the video 
window being pipelined, and times the transfer of data 
through the pipeline. Overlay control circuitry 230 along 
with control circuitry 202, controls the output of data 20 
through output multiplexer 231, including the overlay of the 
video window over the graphics data output through the 

S^Pfc" [back-endl backend 

pipeline 205. A pixel doubler is provided 
to double the number of pixels being generated such that a 
1 280x1 024 display can be driven. 25 
Graphics [back-end] backend 

pipeline 205 includes a first-in/first- 
out memory 232, attribute controller 233, and color look-up 
tabic 234. Each 32-bit word output from graphics FIFO 232 
is serialized into either 8-bit, 16-bit or 24-bit words. The 
8-bit words, typically composed of an ASCII code and an M 
attribute code, are sent to attribute controller 233, When 
16-bit and 24-bit words, which are typically color data, are 
serialized, those words are sent directly -overlay controls 
230. Attribute controller 233 performs such tasks as blinking 
and underlining operations in text modes. The eight bits 35 
output from attribute controller 233 are pseudo-color pixels 
used to index CLUT 234. CLUT 234 preferably outputs 
24-bit words of pixel daia to output multiplexer 231 with 
each index. When video data is being pipelined through 
graphics backend pipeline 205 from the on-screen memory, 40 
CLUT 234 is bypassed 
The eight bit pseudo-color pixels [are] output from attribute 
controller 233 are also sent to overlay controls 230. In the 
preferred embodiment, data is continuously pipelined from 
on-screen memory through graphics [back-end] backend 

pipeline 205 45 
to the inputs of output multiplexer 231. Window data from 
off-screes memory however is only retrieved from memory 
and pipelined through video backend pipeline 204 when a 
window is being displayed In other words, when a window 
has been reached, as determined by control bits set by CPU so 
101 in VW control registers 222, video window display 
controls 222 generate addresses to retrieve the correspond- 
ing data from the off-screen memory space of frame buffer 
107. Preferably, video FIFOs 223 and 224 are filled before 
the raster scan actually reaches the display window such that 55 
the initial' pixel data is available immediately once the 
window has been reached In order to insure that graphics 
memory data continues to be provided to graphics [back-end] backend 
pipeline 205, video window display controls 222 "steal" 
page cycles between page accesses to the graphics memory. 60 
It should be noted that once the window has been reached 
the frequency of cycles used to retrieve window data 
increases over the number used to fill the video FIFOs when 
outside a window. When the frequency of window page 
accesses increases, video window display controls 222/ 65 
arbiter 221 preferably "steal" cycles from page cycles being 
used to write data into the frame buffer. 
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FIG. 3 is a more detailed functional block diagram 
emphasizing the circuitry controlling the overlay of data 
from graphics pipeline 205 with window data from video 
pipeline 204. As discussed briefly above, the inputs to output 
5 multiplexer 231 are data from video [back . en d] backend 

. pipeline 204 

(pixel doubler 237), 16 or 24-bit color data directly from 
graphics back-end pipeline 205 seriaiizer 236 and 24-bit 
color data from the color look-up table 234. The output of 
data to DAC 210 through output multiplexer 231 is con- 
10 trolled by a latch 301 clocked by the video clock (VCUC). 
The remaining circuitry shown in FIG. 3, which will be 
discussed in further detail below, provide the necessary 
control signals to the control inputs of output multiplexer 
231 to select between the video and graphics pipelines. 
t5 The graphics pseudo-pixels output from attribute control- 
ler 233 and the 16-bit or 24-bit graphics or video data output 
directly from seriaiizer 236 are provided to the inputs of 
color comparison circuitry 302. Also input to color com- 
parison [circuit] circuitry 

302 are 16 or 24-bit overlay color key bits 
20 stored in overlay color key register 303. Overlay color key 
register 303 resides within the address space of, and is 
loaded by, CPU 101. Depending on the mode, color com- 
parison circuitry 302 compares selected bits from the over- 
lay color key register 303 with either the 8 bits indexing 
25 look-up table 234 in the color look-up table mode (pseudo- 
color mode) or the 16-bits (24-bits in the alternate embodi- 
ment) passed directly from seriaiizer 236. It should be noted 
that in the illustrated embodiment, overlay color key register 
303 holds [24-bits of] 24 

of overlay color key bits, eight each for 

30 red, green, and [blue/indexl blue index 

comparisons. The specific overlay 
color key bits compared with the input graphics data are 
provided in Table I: 
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MODE 


OVERLAY COLOR KEY BITS COMPARED 


CLUT 
Index 

5:5:5 
5:6:5 
8:8:8 


Rak4:Q> Grcca<4^> 
Rcdc4:Q> Green<5:0> 
RttK7:0> Grcea<7:0> 


Blue/Index 

<3:Q> 
Bbe<4:a> 
Blue<4:0> 
Blue<7:0> 



As shown in FIG. 4A, a first embodiment of color 
comparison circuitry 303 performs the comparisons set forth 
in Table I as a set ofXNOR operations in series with an AND 
4, operation. FIG. 4A depicts first comparison circuitry 400 for 
comparing the 8-bits of graphics pixels received in the 
look-up table mode from attribute controller 233 with the 
8-bit blue/index overlay key bits being held in overlay key 
register 303. Second comparison circuitry 401, performs the 
required comparisons of Table I for the 16-bit data or 24-bit 
received from seriaiizer 236, in, either a 5:5:5, 5:6:5, or 8:8:8 
format An overlay register 402 includes a bit loaded by 
CPU 101 which is used by a selector 403, depending on the 
mode, to select for output, either the result of the compari- 
sons being made by comparison circuitry 400 in the color 
look-up table mode or the results of the comparisons being 
made by comparison circuitry 401. In the illustrated embodi- 
ment, color comparison circuitry 303 processes data on a 
pixel-by-pixel basis and is resynchroniied with both the 
go graphics [back-end] backend 

pipeline 205 and the video back-end 
pipeline 204 by having its outputs latched to the video clock 
(VCLK) by latches 404. 

The output of color comparison circuitry 303 is passedU) 
the "K" control input of overlay control rnuluplexer 3W. 
65 The "P" control input to multiplexer 304 is provided from 
pixel position comparison circuitry 305. The ***** £ 
multiplexer 304 are coupled to an 8-bit overlay OP Code 
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(OOC) register 306. The output of multiplexer 304 is used 
as one control input to output multiplexer 304, which along 
with a single bit set by CPU 101 into output control register 
307, selects which of the data received at the data inputs of 
multiplexer 231 will be output to DAC 210. 5 

Pixel posnjon comparison circuitry 305 includes three 
inputs coupled respectively to video window 1 position 
control circuitry 308, CRT position control circuitry 309 and 
video window 2 position control circuitry 310. In the illus- 
trated embodiment, CRT position controller 309 is located L0 
within CRT controller 220 while video window 1 position 
control circuitry and video window 2 position control cir- 
cuitry 310 are located within video display window controls 
222 (FIG. 2V CRT position control circuitry 309 includes 

counters fw . . . , , 
(wnjch,| which 

track the position of the current pixel being {S 
generated for display. In the preferred embodiment, CRT 
position control circuitry 309 includes at least an x-pqsition 
counter which tracks the generation of each pixel along a 
given display line and a y-position counter which tracks the 
generation of each display line in a screen. The x-posiuon ^ 
counter may for example count pixels by counting each 
VCLK period between horizontal synchronization signal 
(HSYNC) controlling display unit 106. The y-position 
counter may for example count each HSYNC signal occur- 
ring between each vertical synchronization signal (VSYNC) ^ 
controlling the screen generation on display unit 106. FIG. 
4B is an alternate embodiment of the color comparison 
circuitry of FIG. 3. In a first mode* 8 bits [are] 

from attribute 

controller 233 are passed through multiplexer to comparator 

406. Comparator 406 compares the received eight bits with - 
an 8-bit color key in color key register 408; when the 
received 8-bits equal the 8-bit key 1, the output of compara- 
tor 406 goes active (high). In the first mode, control signal 
I6BITGR is high (and the output of NOR gate 409 is 
consequently high) and an active output from comparator 35 
406 is gated through AND gate 410. The output of AND gate 

410 is passed to AND gate 411 and gated with the output 
from the pixel comparison circuitry 305. The output of AND 
gate 411 goes directly to the "B" control input of selector 
231 (in this embodiment multiplexer 304 and register 306 „ 
are eliminated). Thus, when the 8-bit graphics pixels output 
from attribute controller 233 of graphics backend 205 
matches the 8 -bit color key 1 and the window has been 
reached as oetermined by pixel comparison circuitry 305, 
the pixel data output from video backend 204 are passed 4J 
through selector 231. 

In a second mode, 16 bits are received from serializer 236. 
The eight LSBs are passed through multiplexer 405 to 
comparator 406 and the eight MSBs passed to comparator 

407. Control signal 16BITNG is set high. When the LSBs 50 
equal key 1 in color register key 408 and the 8 MSBs equal 
key 2 in color key register 408, the outputs from compara- 
tors 406 and 407 are active (high). The output of AND gate 

411 then goes high when the output from pixel comparison 
circuitry 305, which is coupled to the "B" control input of 55 
selector 231, goes high. Thus, when the 16-bit pixel data 
output from serializer 236 of graphics backend 205 matches 
the 16-bit color key (keys 1 and 2) and a window has been 
reached, the output pixel data from video backend 204 are 
passed through selector 231. w 

FIG. 5 is an expanded functional block diagram of the 
video window position control circuits 308 and a corre- 
sponding portion of the gating of pixel position compare 
circuitry 305. Each position control circuit 310/312 is 
coupled to a screen position x-register 500 and a screen 65 
position y-register 501, and includes a screen x-position 
counter 502, and a screen y-position counter 503. In the 
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preferTet^^Bdiment, registers 500 and 501 arc located 
within vit^^indow control registers 203. For the window 
corresponding to the given video window control circuitry 
308 or 310, registers 500 and 501 are loaded with a value 
representing the x and y screen position of the pixel in the 
upper left comer of that window (the starting pixel). Screen 
x-regiscer 500 and screen y-register 501 in the preferred 
embodiment are loaded fay CPU 101. The screen x-position 
counter 502 counts down from the value held in screen 
x-register 500 with each video clock when P is high for each 
display line and resets with each display horizontal synchro- 
nization signal (HSYNC) (Note that when P is high the CRT 
count matches the position count). Screen y-position counter 
503 counts down from the value set into screen y-register 
501 for each horizontal sync signal (HSYNC) at the start of 
each display line and resets with each VS YNC at the start of 
each new screen (The position counters are allowed to count 
only when they match their perspective CRT). The counts 
values in the counters of CRT position control circuitry 309 
are compared pixel by pixel with the counts in screen 
x-position counter 502 and screen y-position 503 of each 
video window position control circuitry 308 and 310. When 
both the x and y counts in the counters of CRT position 
control circuitry 309 match the corresponding x and y counts 
in respective counters 502 and 503 of either video window 
control circuitry 308 or 310, the control signal P to multi- 
plexer 304 is activated The activation of control signal P 
indicates that the raster scan on display 106 has reached the 
position of a pixel within the [window] window, 

. and data from video 
pipeline 205 may be painted depending on the value being 
held in overlay OP Code (OOC) register 306 and the K 
35 control inputs to multiplexer 304. 

A 4-bit OP Code loaded by CPU 101 into overlay OP 
Code register 306 in conjunction with the control signals 
applied to the "F* and "K" control inputs to multiplexer 304 
4Q control the presentation of an active (assumed high in the 
illustrated embodiment) control signal to the "B" control 
input to output multiplexer 23t The other ("A") input to 
output multiplexer 231 receives a bit from overlay mode 
register 402 (FIG. 4) t as loaded by CPU 101. In the 
45 illustrated embodiment, the selection between the streams 
from the graphics and video backends at the 0,1,2 inputs to 
output multiplexer 304 in response to the signals presented 
at the corresponding control inputs "A" and "B w is in 
accordance with Table O: 
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Control Iflpui A Control Input B Selected Stream 



0 


0 


Graphics or 






video pixel* 






from gnpfaici 






pipeline 205 


I 


0 


Graphics pixeli 






from CLUT 23* at 






input 1 


c 


1 


Video or 


1 


1 


gr&phics from 






video backead 






204 



The OP Codes used in the illustrated embodiment, the 
65 effective overlay and the corresponding inputs to the control 
inputs of multiplexer 304 are listed in Table m (active state 
is assumed{:]); 



Overlay Op Code Multiplexer 307 
Register 309 Value Control Inputs 



0 



N/A 



Pixels passed 
only from 
graphics pipeline 



A 



C 



s 



Inputs K and P 
active 



[npul P acuve 



Inputs K active 



205 

Paint pixels from 
video backend 
pipeline 204 only 
inside video 
window VDW„ 
Paint from video 
backend pipeline 
204 window VDW B 
when color key 
matches 

Paint from video 
backend pipeline 
204 if color key 
matches 
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In the illustrated embodiment, if a Oh is written into OOC 
register 306 by CPU 101, only pixels from graphics pipeline 
205 arc pipelined through multiplexer 304. In this case any 
signals applied to the P and K control inputs to multiplexer 
304 have no effect (i.e., will not result in a high output from 
multiplexer 304). In the illustrated embodiment, if an Ah is 
written into OOC register 306, pixels from video pipeline 

204 will be passed to DAC 210 only when pixel position 
comparison circuitry 305 determines that the raster scan has 
reached a pixel in the window and hence the control signal 
going to the P input of multiplexer 304 has been activated. 30 
If on the other hand, an 8h is written into OOC register 306, 
data is passed through output multiplexer 231 to DAC 210 
when pixel position comparison circuitry 305 determines 
that the raster scan has reached a pixel on the display screen 
within the window and color compare circuitry 302 has 35 
determined that the incoming data from graphics pipeline 

205 matches the overlay color key held in overlay color key 
register 303. In this case, the data from video pipeline 204 
is passed to DAC 210 when both the P and the K inputs to 
multiplexer 304 are active. Finally, when an OpCode of C is 40 
programmed into OOC register 306, data from video pipe- 
line 204 is passed if the incoming data from graphics 
pipeline 205 matches the overlay color key held in overlay 
color key register 303. In this case, the activation of the K 
control input activate the output of multiplexer 304 to switch 45 
the input of multiplexer 231 to pass the corresponding video 
pixels. 

Display control circuits embodying the principles of the 
present invention have substantial advantages over the prior 
art In particular, output control circuits built in accordance 50 
with the principles of the present invention allow for the 
flexible display of both graphics and video on the same 
screen. In particular, pixel position comparison circuitry 305 
along with video window position control circuits 308 and 
310 and CRT position control circuitry 309 allow for [a ] 

one ss 
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Although the present invention and its advantages have 
been described in detail, it should be understood that various 
changes, substitutions and alterations [ an j can 

be made herein 

without departing from the spirit and scope of the invention 
as defined by the appended daims. 



«i COLUMN 14 CONTIIVUED 
fs claimed is: 

1. A single integr at epigraphies and video controller* 
adapted for driving a display < seqU entiatly 

-comprising: 

an interface for receiving words of pixel data* each said 
word associated with an address buffer; 
10 circuitry for writing each said word of said pixel data 
received by said interface to a one of on-screen and 
off-screen memory areas of a frame buffer, 
circuitry for selectively retrieving' 
as data is provided for display, 

said words from said 

on-screen and off-screen areas; 
15 a first pipeline for substantially continuously 

processing words of graphics data 
retrieved from said frame buffer* and 
a second pipeline for processing words of video data 
retrieved from said frame buffers o that the 

video data is ready for display once a display raster 
scan reaches a display position o£ a window. 

2. The controller of claim 1 and further comprising output 
20 selection circuitry for selecting for output between graphics 
data received from said first pipeline and data received from 
said second pipeline, said selection circuitry operable to: 
in a first mode, pass data from said 

[first pipeline:] first pipeline; m0 \ 
^ in a second mode, pass data from said second pipeline 
when said data corresponds to a selected display posi- 
tion of a display window. 
{F*s 3. The controller of claim 2 wherein said selection cir- 

cuitry is further operable to: 
:f*f- 30 ^ a third mode, pass data from said second pipeline when 

said data corresponds to said selected display position 
^■|| of said display window and data from said first pipeline 

?£= ' t match a color key. 

* 4. The controller of claim 3 wherein said selection cir- 
isM 35 cuitry is further operable in a fourth mode to pass data from 

said second pipeline when data from said first pipeline 
£^ match a color key. 

5. The controller of claim 1 wherein said circuitry for 
s retrieving maintains a stream of graphics data to said first 

40 Pipeline and provides video data to said second pipeline 
I*. when a display raster scan reaches [said display] a display 

|F S [said window] a window. position of 

6. The controller of claim 1 and further comprising: 
Jg a video port for receiving real-time video data; and 

45 circuitry for generating an address to said memory at 
^ which said real-time video data is to be stored. 

7. The controller of claim 1 wherein said second pipeline 
includes a first first-in-first-out memory for receiving data 
for a first display line of pixels in memory and a second 

50 first-in-fixst-out memory for receiving data from a second 
display line of [pixels memory] pixels in memory. 

8. The controller of claim 7 wherein said first display 
[line adjacent] line is adjacent 

in memory to said second display line. 

9. The controller of claim 7 

further comprising output selection circuitry 

wherein said output selection 

55 circuitry comprises: 

an output selector for selecting between data from said 
second pipeline and data from said first pipeline in 
response to a selection control signal; 

^ a register for m ai n taining a plurality of overlay control 
bits; 

window position control circuitry for selectively gener- 
ating a position control signal when a word of said data 
stream from said second pipeline fails within a display 
65 window; 

color comparison circuitry for comparing words of said 
data stream from said first pipeline with a color key and 
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for providing in response a color comparison control 
signal; and 

a comrol selector for selectively providing [» 1 ^ d 

selection 

control signal in response to said overlay control bits in 
said register and at least one of said position control and 5 
color comparison control signals. 

10. The controller of claim 9 wherein said window 
position control circuitry comprises: 

window position counters operable to increment from 
initial count values corresponding to a starting pixel of 10 
a display window as data representing each pixel in a 
display screen is pipelined through said overlay control 
circuitry; 

screen position counters operable to count as data repre- 
senting each pixel in said display screen is pipelined 15 
through said overlay control circuitry; and 

comparison circuitry operable to compare a current count 
in said window position counters and a current count in 
said screen position counters and selectively generate 
said position control signal in response. 20 

11. The controller of claim 9 wherein said color compari- 
son circuitry comprises; 

a color key register for storing bits composing said color 
key; and 

a plurality of AND-gates for comparing said words of said 25 
graphics data stream with bits of said color key. 

12. The controller of claim 1 wherein said interface 
includes a dual-aperture port 

13. A single integrated controller comprising: 
Ci?cuitry for writing selectively, on a word by word basis, 

r each word of received 30 

data into [s] a 

selected one of on-screen and off-screen 
memory spaces of a frame buffer, 
a first port for receiving video and graphics data, a word 
of said data received with an address of said memory 35 
spaces directing said word to be processed as a word of 
video data or a word of graphics data; 
a second port for receiving real-time video data; 
circuitry for generating an address associated with a 
selected one of said memory spaces for a word of said 40 
real-time video data; 
circuitry for selectively retrieving said words of data 
on a word by word basis .from 

said on-screen and off-screen memory spaces as data is 

rastered for driving a display in a sequential f ashiorw 

a graphics backend pipeline for processing ones of said 
words of data representing graphics data retrieved from 
said frame buffer, 

a Video backend pipeline , separate 

from said graphics backend pipeline, 

, for processing other ones of said 
words of data representing video data retrieved from 50 
said frame buffer, said circuitry for retrieving always 
rastering a stream of data from said frame buffer to said 
graphics backend pipeline and rastering video data to 
said video backend pipeline 1 [when] so that the 

video data is ready for display once 

a display raster scan 
reaches a display position of a window; and 55 
output selector circuitry for selecting for output between 
words of data output from said graphics backend pipe- 
line and words of data output from said video backend 
pipeline. . 

14. The controller of claim 13 wherein said output selec- « 

tor circuitry ... * 

is further operable to select between graphics data output 
from a color look-up table and true color data output from 
said graphics pipeline. 

15. The controller of claim 13 wherein said output selec- 
tor circuitry 

is operable to: 65 
in a first mode, pass only a word of data output from said 
graphics pipeline; 
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in ?BRnd mode, pass a word of data output from said 
video pipeline when said display raster scan has 
reached a display position corresponding to a window 
and a word of data from said graphics pipeline when 
5 said display raster scan is in any other display position; 
in a third mode, pass a word of data output from said 
video pipeline when said display raster scan has 
reached a display position corresponding to a window 
and a corresponding word of data from said graphics 
10 pipeline matches a color key and a word of data from 
said graphics pipeline when said display raster scan is 
in any other display position; and 
in a fourth mode, pass a word of data from said video 
pipeline when said corresponding word of data from 
15 said graphics pipeline matches a color key and a word 
of data from said graphics pipeline when said [display 

raster scan is in any other display position] corresponding 
word does not match said color key. 

16. The controller of claim 13 wherein said video pipeline 
includes a first first-in-first-out memory for receiving a 
plurality of words of data for a first display line of pixels in 

20 memory and a second first-in-first-out memory " 
[or] for receiving 

a plurality of words of data IfromJ for 

a second display line of 

pixels in memory. 

17. The controller of claim 16 wherein said first display 
line is stored adjacent in memory to said second display line. 

25 18. The controller of claim 13 wherein said output selec- 
tor circuitry comprises: 
a control selector having a plurality of control inputs 
coupled to a register, said register storing a plurality of 
3Q overlay control bits; 

window position control circuitry coupled to a first con- 
trol input of said control selector, said window position 
control circuitry operable to selectively provide a first 
control signal to said first control input when a word of 
35 data being pipelined through said video pipeline falls 
within a display window; 
color comparison circuitry operable to compare a word of 
data being pipelined through said graphics pipeline 
with a color key and provide in response a second 
40 control signal to a second control input of said control 
selector, and 

wherein said control selector is operable to provide an 
output selection control signal in response to at least 
one of said first and second control signals and said 
45 overlay control bits being stored in said register. 

19. The circuitry of claim 18 wherein said output selector 
circuitry further includes a third control input coupled to 
certain bits of said graphics pipeline, said output selector 
circuitry further 

operable to select between data on said respective 
50 video and graphics pipelines in response to said certain bits 
presented to said selector circuitry. 

20. The circuitry of claim 18 wherein said window 
position control circuitry comprises: 

a window x-position counter operable to count from a 
55 loaded x-position value in response to a video clock, 
said x-position counter reloading in response to a display 
horizontal synchronization signal; 
a window y-position counter operable to count from a 
loaded y-position value in response to said horizontal 
60 synchronization signal, said y-position counter reload- 
ing in response to a display vertical synchronizanon 
signal; . . 

CRT position circuitry operable to main^m ^ 

63 sponding to a current display pixel; and 

comparison circuitry operable to compare current counts 
in said window counters with said current counts held 
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in said CRT position circuitry and generate in response 
said first control signal. 

21. The circuitry of claim 20 wherein said window 
position control circuitry further comprises an x-position 
register for holding said x-position value for loading into 
said x-position counter and a y-position register for holding 
said y-position value for loading into said y-position 
counter. 

22. The circuitry of claim 13 wherein said color compari- 
son circuitry comprises: 

a color key register for storing a plurality of color key bits; 
and 

a plurality of XNOR-gates each having at least one input 
coupled to a bit position in said color key register and 
at least one input coupled to said graphics data path. 15 

23. The circuitry of claim 13 wherein said video pipeline 
comprises: 

a first-in/first-out memory for receiving a first stream of 

words of data from said frame buffer, 
a second first-in/first-out memory disposed in parallel 20 

with said first first-in/first-out memory for receiving a 

second stream of words of data from said frame buffer; 

and 

interpolation circuitry for selectively generating an addi- 
tional word of data by interpolating a word of said first 25 
stream and a word of Said Second 

stream data output from 
said first and second first-in/first-out memories. 

24. The controller of claim 13 wherein said first [pore] port 
comprises a dual-apertuxe port. 

25. A display system comprising: 30 
a first backend pipeline for processing data; 

a second backend pipeline for processing graphics data 
disposed in parallel to said first processing pipeline; 

a multi-format frame buffer memory having on-screen 
and off-screen areas each operable to 

to allow said frame buffer to 

simultaneously 
store data in graphics and video formats; 
[a] an 

input port for receiving both graphics and video data, 
each word of said data associated with an address 
directing said word to be processed as either graphics ^ 
or video data; 

circuitry for writing a word of said playback data into a 
selected one of said areas of said multi-format memory; 

memory control circuitry for controlling the transfer of 
data between said first backend pipeline and said frame 45 
buffer and between said second backend pipeline and 
said frame buffer, 

a display unit; and 

overlay control circuitry for selecting for output to said ^ 
display unit between data provided by said first back- 
end pipeline and data provided by said second backend 
pipeline. 

26. The display system of claim 25 wherein said second 
backend pipeline includes: 55 

a first first-in-first-out memory for receiving first selected 
data; 

a second first-in-first-out memory disposed in parallel to 
said first first-in-first-out memory for receiving second 
selected data; and 60 
interpolation data for generating additional data by inter- 
polating data output from said respective first and 
second first-in- first-out memories. 

27. The display system of claim 26 wherein said second 
backend pipeline further comprises color conversion cir- 65 
cuitry for converting data received from said frame buffer in 

a video format to a graphics format. 
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28. The display system of claim 25 [and] 

further comprising 

a video front-end pipeline for inputting video data into a 
selected one of on-screen and off-screen spaces of said frame 
buffer comprising: 

a video data port for receiving video data from a real time 

data source; and 

input control circuitry for receiving framing signals asso- 
ciated with said real time data and generating corre- 
sponding addresses to said selected one of said spaces 
in response. 

29. The display system of claim 28 wherein said video 
front-end pipeline further comprises encoding circuitry for 
packing said video data prior to storage in said selected one_ 

of said 

spaces. 

30. The display system of claim 28 wherein said video 
front-end pipeline further [comprising] comprises 

, multiplexing circuitry 
for selecting between video data received through said video 
data port and data received from f sa jd] a dual 

fJOt ^ *f* y A SyS L tCm of claim 30 whereXd^ideo 
c^v^nf * ^ ft^cwpri" conversion circuitry for 
converting graphics data received through said dual-aperture 

™ a 60 f0nnat f0r Stora * e m said selec *d one ofsaid 
spaces. 

hJ+L? 6 ' d r Play SyStCm of cIaim 25 wherc * said first 
backend pipeline processes playback video 

*y stcm <* da** 25 wherein said input port 
composes a dual-aperture inout port. 

34. A single integrated 

display data processing system comprising: 
circuitry for writing data into an on-screen space of a 
frame buffer, 

circuitry for writing data into an oflF-screcn space of said 
frame buffer; 

a video pipeline for processing data output from a selected 
one of said on-screen and off-screen spaces comprising: 
a first first-in- first-out memory for receiving selected 

data from said selected space; 
a second first-in-first-out memory disposed in parallel 
to said first first-in-first-out memory for receiving 
: other selected data from said selected space; and 
an interpolator for generating additional data by inter- 
polating data output from said respective first and 
second first-in-first-out memories; 
a graphics pipeline disposed in parallel to 

and separate from 

.said video 

pipeline for processing data output from a selected one 
of said on-screen and off-screen spaces; and 
an output selector for selecting between data output from 
said video pipeline and data output from said graphics 
pipeline. 
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35. The system of claim 34 [and] 



further comprising 



selection control circuitry for generating an output control 
signal for controlling said output selector comprising: 
a control selector having a plurality of data inputs coupled 
to a register, said register for storing a plurality of 
overlay control bits; and 
color comparison circuitry operable to compare bits of 
data output from said graphics pipeline with a color key 
and provide in response a control signal to a control 
input of said control selector 



36. The system of claim 34 [and] casing 



50 window position control circuitry operable to provide a 
second control signal to a second control input of said 
control selector when data from said video pipeline falls 
within a display window. 

37. A' single integrated 

'display controller comprising: 
65 circuitry for selectively retrieving data from an associated 

multi-format frame buffer, the frame buffer having 
separate storage locations respectively operable for allowing 



•for simultaneously 



[snoring] storing 
graphics and video data; 



tiir. said frame buffer; 
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a first pipeline for processing words of graphics data 

selectively retrieved from said frame buffer; and 
a second pipeline % sepa rate from the first pipeline, 

for processing words of video data 
selectively retrieved from said frame buffer. 

38. The controller of claim 37 wherein said first and 5 
second pipelines are disposed in parallel and concurrently 
process data. 

39. The controller of claim 38 l«w*J 

further comprising 

output selection circuitry for selecting for output between 
graphics data received from said first pipeline and video data 
received from said second pipeline. 

40 The controller of claim 37 wherein said frame buffer 
is partitioned into on-screen and off-screen areas, each of 
said on-screen and off-screen areas operable to 

/ allow the buffer to 

simulta- 
neously store both graphics and video data. 15 

41 The controller of claim 37 wherein said circuitry for 
selectively retrieving is operable to retrieve a substantially 

constant 

stream of graphics data from said frame buffer and provide 
said stream of graphics data to said first pipeline. 

42 The controller of claim 41 wherein said circuitry for 
selectively retrieving is operable to retrieve at least one said 
word of video data from said frame buffer and provide said 
at least one word of said video data to said second pipeline, 
only when said display controller is generating a video ^ 
display window, 

43 A display controller for interfacing a multi-format 
frame buffer and a display device, the multi-format frame 

buffer having on-screen and off-screen areas each operable for allowing 

simultaneously storing both graphics and video pixel data in the frame buffer, 

said 

. . - 30 

display controller comprising: 
circuitry for selectively retrieving pixel data from a 

selected one of said on-screen and off-screen areas of 

said frame buffer, 
a graphics backend pipeline for processing graphics data 35 

retrieved from said selected one of said areas of said 

frame buffer, 



20 

a video backend pipeline for processing video data 
retrieved from said selected one of said areas of said 
frame buffer, and 
5 an output selector for selectively passing to said display 
device data received from said graphics or video back- 
end pipelines. 

44. The display controller of claim 43 wherein said 
circuitry for selectively retrieving is operable to retrieve at 

10 least one said word of video data from said frame buffer and 
provide said at least one said word of video data [no] to 

said 

second pipeline only when said display controller is gener- 
ating a video display window. 

45. The display controller of claim 43 wherein said output 
selector is operable to: 

in a first [modes] mode 

pass data from said graphics pipeline; and 
in a second [modes] mode 

pass data from said video pipeline 
when a display position corresponding to a video 
20 display window has been reached. 

46. The display window of claim 43 wherein said output 
selector is operable to: 

in a first mode, pass data from said graphics pipeline; and 
23 " in a second mode, pass data from said video pipeline 
when a display position corresponding to a video 
display window has been reached and data from said 
graphics pipeline match a color key. 

47. The display controller of claim 43 wherein said output 
30 selector is operable to: 

in a first mode, pass data from said graphics pipeline; and 
in a second mode, pass data from said video pipeline 
when data from said graphics pipeline matches a color 
3J key. • 

48. The controller of claim 42 in which graphics data is substantially ' 
continuously retrieved for a display while video data is retrieved only for 
the video display window, 
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ABSTRACT 



A graphics and video controlJer 105 is provided which 
includes a dual aperture interface 206 for receiving words of 
graphics and video pixel data, each word of such data 
associated with an address directing that word to be pro- 
cessed as either graphics or video data. Circuitry 200, 201, 
202, 207, 208 is provided for writing a word of the pixel data 
received from the interface 206 to a one of the on- and 
off-screen memory areas corresponding to the address asso- 
ciated with the received word. Circuitry 201, 202 is provided 
for selectively retrieving graphics and video data from the 
on-screen and off-screen memory areas. A first pipeline 205 
is provided for processing data received from the on-screen 
area of frame buffer 107 while a second pipeline 204 is 
provided for processing data retrieved from the off-screen 
area of the frame buffer. 
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